One of the great things about the Magic Leap One headset is that compared to mobile AR, your hands are finally free. You no longer need to hold a mobile device in your hand in order to see the virtual content. You are free to experience mixed reality apps in a more natural, comfortable, and immersive way.
This opens up boundaries that were previously closed. Now that you don’t need to actually hold a phone or tablet in your hands, you are free to use your hands for other things.
Of course, Magic Leap won’t leave you to have your hands just resting down won’t they? After all, the goal is to make the mixed reality experience authentic and realistic as possible when applicable. This means, allowing users to not just see digital objects all around them, but interacting with them as well, similar to how you interact with objects in real life. This means, being able to move virtual objects with your hands, picking them up, throwing, pushing and tapping them, etc.
The Magic Leap offers several ways for you to take advantage of your unoccupied hands.
- Use the Magic Leap One 6DoF Control (Controller) to interact with content
- Use the Magic Leap Mobile App as a backup input control option for the Magic Leap One Control (aside from other advanced connectivity functionality like transferring photos and videos, enter text, etc.)
- Use hands tracking to define interactions with your app
How Good it the Magic Leap One Hand Tracking?
In the official documentation, The Controller is referred to by Magic Leap as the primary option, second is the App and third is the hand-tracking feature.
In my opinion hand tracking is the most fascinating option of the three, but Magic Leap mentions that although this option is the most favorite way of interaction, it’s mostly for short durations. The reason for that is because users will need to raise their arm for their hands to be in the detectable areas where their gestures can be read and raising the arms and keeping the hands up is tiring.
The official recommendation is to “avoid over-using these capabilities”. However, this doesn’t mean that you always follow that recommendation. What thing for sure, putting and keeping your hands up is easier to do without having two controllers in our hands, like you have in VR.
Let’s say you want to develop a mixed reality spellcasting game. I want to play a spellcasting game, I’ve played some in VR like Tales of Glacier, The Wizards, The Mage’s Tale, inVokeR, Chaos Edge, Runes: the forgotten path and many others and I really want to be able to play a similar game in MR. I’ve also played HADO, which was played hands-free using an AR headset. If you want to create this type of gameplay experience, the best way to do it is by using hand gestures. Imaging seeing a fireball coming out of your hand or energy wrapping around your arm before you release a devastating spell attack.
When playing these type of game, it’s OK to choose to have a lot of hand-arm movement. Having said that, there might be some things that will impair the use of this functionality in your game, like the relatively small FOV and the fact that your hand need to be seen by the device’s Gesture FOV, the fact that the more hand postures your app needs to recognize, the worse recognition gets and this also negatively affects the performance of the game (according to the official documentation).
In VR, the 6DoF motion controllers are designed and optimized for these type of interaction. They are very accurate, they can be used with much fewer limitations and they built with great ergonomics, so users can use them for a long period of time.
Magic Leap One can tell the difference between the left and right hand. The Magic Leap One hand tracking system can recognize the position of 8 key points on the hand for every one of its recognized hand poses.
The Magic Leap One supports numerous gestures but those are pre-defined ones and I haven’t come across anything about the documentation regarding custom hand gestures.
The Tonandi app was actually a good showcase of some hand interaction. Some seem to be based on pre-defined gestures, others are based on the hand and finger location. Still, it seems kind of sparse and inaccurate compared to other hand tracking technologies that I’ve seen.
Hand gestures are a big deal in mixed reality and many companies are working the perfect this. Here, take a look at this amazing Project North Star demo and Leap Motion: Orion hand tracking software just to get a glimpse of what’s possible. It seems as, at the moment, the hand tracking is not the strongest part of Magic Leap One. Leap Motion is one of the leading companies in the hand-tracking field. After seeing their hand-tracking demo videos, it made me wonder why Magic Leap didn’t bring such advanced functionality to the Magic Leap One, seems like a missed opportunity, unless I missed something.
In my mind, I see hand tracking is a key interaction component in many future mixed reality apps and of course games. I mean, raising the hand to use a controller is also a physical effort.
You know, Leap Motion can be embedded directly into any VR/AR headset, which makes you wonder if such an advanced hand-tracking technology can be used with the Magic Leap One as well. However, as a mobile wearable device, the optimal way is to have all those hardware-based technologies self-contained in the headset itself.
Reading the documentation of the Magic Leap One’s hand tracking and comparing it with the 180°x180° tracking super accurate Leap Motion hand tracking technology, left me with somewhat bitter taste. Does anyone have any doubts about the importance of this technology? I’m sure many developers would have liked to be able to develop their apps and games with this type of “Leap Motion”-like technology from day one. What’s the point of having your hands free if you don’t take a really good advantage of them. You want to make the virtual mix seamlessly with the real world, don’t forget the human interaction within it.
The Mixed Reality magic works wonders when the users aren’t just observers but an integrated part of the experience, being able to not just see, but interact and preferably also feel (via haptic feedback) the virtual content all around them. Preferably with their hands, where it feels the most intuitive and natural. Being able to manipulate virtual content in many different ways and feel free to do so in a natural arm, hand and finger motions like we do in the real world. This will further help to make the virtual blend more seamlessly with our reality, eventually to the point where both are indistinguishable. A point, where everyone will want to get involved and try those mixed reality experiences themselves. A point where when you put your MR headset on, literally everything is possible, where there is an opportunity for things to behave differently, for you to be able to do things that you could never have done before.
This is how important this feature is. Just look at an object in front of you, maybe it’s a coffee cup, your laptop, or mobile phone. Imagine in your head that this is actually a virtual object, yet it appears real. Now hold it with your hand and lift it up. If we can replicate that feeling and bring it to mixed reality, we have achieved something extraordinary. We are not there yet, but we are not far from it.
Update: I’ve just read Lucas Rizzotto full review of the Magic Leap One Creator Edition on virtualrealitypop.com where talks about hand-tracking and he says that it feels pretty limited compared to other superior forms of tracking like the Leap Motion. So it’s yet another opinion that lead us to an understanding that the Magic Leap One has quite a limited hand-tracking technology.
This is a big topic, and I just wanted you to be aware of it and encourage you to do your own investigation to see what technologies fit your game best. I will continue to do my own research and try to bring you some more interesting insights. After seeing all the amazing hand tracking systems out there, it made me wonder why Magic Leap hasn’t bought any one of those companies and used their advanced technologies. Anyways, there are questions to be answered, hopefully, we’ll get answers to these ones pretty soon.